-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Yash/dev llava next #10749
base: main
Are you sure you want to change the base?
Yash/dev llava next #10749
Conversation
https://wandb.ai/joc/llava_next_energon/runs/70551i0j?nw=nwuserykarnati - fine tuning convergence run on LLaVA dataset |
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: artbataev <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
@@ -0,0 +1,161 @@ | |||
import argparse | |||
import os |
Check notice
Code scanning / CodeQL
Unused import
@@ -0,0 +1,161 @@ | |||
import argparse | |||
import os | |||
import sys |
Check notice
Code scanning / CodeQL
Unused import
import os | ||
import sys | ||
|
||
import requests |
Check notice
Code scanning / CodeQL
Unused import
import requests | ||
import torch | ||
from megatron.core.optimizer import OptimizerConfig | ||
from megatron.energon import VQASample |
Check notice
Code scanning / CodeQL
Unused import
import torch | ||
from megatron.core.optimizer import OptimizerConfig | ||
from megatron.energon import VQASample | ||
from PIL import Image |
Check notice
Code scanning / CodeQL
Unused import
from nemo.collections import llm, vlm | ||
from nemo.collections.multimodal.data.energon import SimpleMultiModalDataModule | ||
from nemo.collections.multimodal.data.energon.config import MultiModalSampleConfig | ||
from nemo.collections.vlm import ImageDataConfig, Llava1_5Config7B, LlavaModel, LlavaNextTaskEncoder |
Check notice
Code scanning / CodeQL
Unused import
from nemo.collections.vlm import ImageDataConfig, Llava1_5Config7B, LlavaModel, LlavaNextTaskEncoder | ||
from nemo.lightning.pytorch.optim import CosineAnnealingScheduler | ||
from nemo.lightning.pytorch.optim.megatron import MegatronOptimizerModule | ||
from nemo.utils import logging |
Check notice
Code scanning / CodeQL
Unused import
# Global and micro batch sizes | ||
gbs = 32 | ||
mbs = 4 | ||
seq_length = 256 |
Check notice
Code scanning / CodeQL
Unused local variable
from nemo.collections import llm, vlm | ||
from nemo.collections.multimodal.data.energon import SimpleMultiModalDataModule | ||
from nemo.collections.multimodal.data.energon.config import MultiModalSampleConfig | ||
from nemo.collections.vlm import ImageDataConfig, LlavaNextTaskEncoder |
Check notice
Code scanning / CodeQL
Unused import
# Global and micro batch sizes | ||
gbs = 128 | ||
mbs = 4 | ||
seq_length = 4096 |
Check notice
Code scanning / CodeQL
Unused local variable
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: cuichenx <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: ashors1 <[email protected]>
Signed-off-by: ashors1 <[email protected]>
* add LinearAdapter Signed-off-by: Alexandros Koumparoulis <[email protected]> * add hf lora example Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove unused imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * subclass mixin Signed-off-by: Alexandros Koumparoulis <[email protected]> * remove stale imports Signed-off-by: Alexandros Koumparoulis <[email protected]> * undo Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix scale Signed-off-by: Alexandros Koumparoulis <[email protected]> * regex selector for peft Signed-off-by: Alexandros Koumparoulis <[email protected]> * move lora Signed-off-by: Alexandros Koumparoulis <[email protected]> * fmt Signed-off-by: Alexandros Koumparoulis <[email protected]> * hf_auto_model_for_causal_lm finetune recipe Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>
Signed-off-by: Abhishree <[email protected]> Co-authored-by: Pablo Garay <[email protected]>
* Add a build option to load_context Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Adding test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * Trying to fix failing CPU test Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> * cherry-pick fix Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Marc Romeijn <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]>
* Move AutoTokenizer inline Signed-off-by: Marc Romeyn <[email protected]> * Move einops to common requirements Signed-off-by: Marc Romeyn <[email protected]> * Move AutoTokenizer import to top-level again in fine_tuning Signed-off-by: Marc Romeyn <[email protected]> * Move megatron init inside nemo.lightning Signed-off-by: Marc Romeyn <[email protected]> * Make megatron_lazy_init_context work when transformer-engine is not installed Signed-off-by: Marc Romeyn <[email protected]> * Only import get_nmt_tokenizer when needed Signed-off-by: Marc Romeyn <[email protected]> * Apply isort and black reformatting Signed-off-by: marcromeyn <[email protected]> --------- Signed-off-by: Marc Romeyn <[email protected]> Signed-off-by: marcromeyn <[email protected]> Co-authored-by: marcromeyn <[email protected]>
* add docs Signed-off-by: stevehuang52 <[email protected]> * update doc and fix missing param Signed-off-by: stevehuang52 <[email protected]> --------- Signed-off-by: stevehuang52 <[email protected]>
* Enable ckpt features by default (async ckpt), ckpt every 15mins and reduce preemption time to 1min Signed-off-by: Shriya Palsamudram <[email protected]> * fix ssm tests Signed-off-by: Shriya Palsamudram <[email protected]> * Make note that ckpt_async_save is disabled for SSMs Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for SSMs with fix Signed-off-by: Shriya Palsamudram <[email protected]> * Disable async ckpt in the peft test as it is a known bug, add note. Signed-off-by: Shriya Palsamudram <[email protected]> * Fix failing unit tests Signed-off-by: Shriya Palsamudram <[email protected]> * Ashors/peft async ckpt (#11010) * [WIP] prototype for supporting async checkpointing with peft Signed-off-by: ashors1 <[email protected]> Signed-off-by: Shriya Palsamudram <[email protected]> * Enable async ckpt for the peft test Signed-off-by: Shriya Palsamudram <[email protected]> * Fix peft setup test Signed-off-by: Shriya Palsamudram <[email protected]> --------- Signed-off-by: Shriya Palsamudram <[email protected]> Signed-off-by: ashors1 <[email protected]> Co-authored-by: ataghibakhsh <[email protected]>
* Mixtral TP8 EP1 Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>
Signed-off-by: Yashaswi Karnati <[email protected]>
Signed-off-by: yashaswikarnati <[email protected]>
Signed-off-by: yashaswikarnati <[email protected]>
Signed-off-by: Yashaswi Karnati <[email protected]>
Signed-off-by: yashaswikarnati <[email protected]>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: pablo-garay <[email protected]>
Signed-off-by: yashaswikarnati <[email protected]>
4ded144
to
accc256
Compare
What does this PR do ?
Support training of LLaVa NeXt model.
Collection: [Note which collection this PR will affect]
Multimodal
Changelog
Usage
The only change is to use the energon based data module as shown below with existing NeVA model
GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information